Analysis of Optimization Algorithms via Sum-of-Squares

We introduce a new framework for unifying and systematizing the performance analysis of first-order black-box optimization algorithms unconstrained convex minimization. The low-cost iteration complexity enjoyed by renders them particularly relevant applications in machine learning large-scale data analysis. Relying on sum-of-squares (SOS) optimization, we hierarchy semidefinite programs that give increasingly better convergence bounds higher levels hierarchy. Alluding to power SOS hierarchy, show (dual the) first level corresponds estimation problem (PEP) introduced Drori Teboulle (Math Program 145(1):451–482, 2014), powerful determining rates algorithms. Consequently, many results obtained within PEP can be reinterpreted as degree-1 proofs, thus, provides promising approach certifying improved means higher-order certificates. To determine analytical rate bounds, this work, use derive noisy gradient descent with inexact line search methods (Armijo, Wolfe, Goldstein).